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ABSTRACT 



This paper discusses a design methodology that would allow 
schools and colleges to identify the salient and tangible practices and 
processes that they will use to define their programs, explaining how to make 
a powerful technology of evaluation part of the design. The paper emphasizes 
the need to bring greater clarity to the core competencies and expected 
outcomes of programs in order to make evaluation purposeful. Five key steps 
are presented, using examples derived from a longitudinal school reform 
project: (1) establish the pedagogical and curricular core competency of the 

school or teacher education program; (2) build a curriculum model around the 
core competencies; (3) support the model with organizational design, building 
those structures that support the growth of faculty with the core competency; 
(4) use technology to enable reform by translating the essential elements of 
the reforms into practical tools that teachers, students, and parents can 
use; and (5) employ principles of effective practice to connect the teaching 
and learning processes. (Contains 5 figures and 18 references.) (SM) 
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The current backlash in education surrounding the accountability and standards 
movements (e.g., Houston, 1999) creates an interesting context within which to take a 
position on the identification of educational evaluation measures. The reasons for the 
backlash are consistent with the long history in education of fixing the parts while 
ignoring the whole. We have known for some time that educational systems and 
individual organizations are remarkably complex, frustratingly informal in their 
professional culture, and highly resistant to change (Evans, 1996; Sarason, 1990). As 
such they are ever ready to rebuff one dimensional innovation efforts. Whether it be the 
international trend toward devolving responsibility without providing the necessary 
expertise to make devolution work; providing access to technology without curriculum 
integration training, or setting new standards for teachers without engaging in the 
fundamental reform of models of preparation, we know the outcome all too well. In each 
case we see the same predictable and repeated pattern of failure. 

So much of our accreditation and evaluation effort is disproportionately focused on the 
correlates of school success (e.g., adequacy of infrastructure, and the articulation of 
curriculum considerations as opposed to evidence of student outcomes). Far less 
emphasis is placed on those things that should be causally connected to the learning of 
students, largely because they are not formally established in the design of educational 
organizations. By way of comparison when accreditation or accountability standards 
move too rapidly toward a focus on student outcomes, their proponents face the opposite 
problem — educational organizations that have not evolved sufficiently to deliver those 
outcomes. 



The Problem (With Pictures') 

OPTION A: In education we reform this way: 




The problem with this picture is the parts never seem to arrive at the same time, nor do 
they fit together. We provide access to technology without a parallel effort to train and 
support adequately. The technology is underutilized despite the incredible expense 
required to put it in place. This type of reform results in a failure to make demonstrable 
improvement in the core activity of schools — student learning. The result is a stream of 
initiatives that orbit the periphery of educational systems and rarely penetrate the core of 
individual organizations nor the classrooms of individual schools, making legitimate 
evaluation exceptionally difficult (Elmore, 1996; Honig, 1994; Pogrow, 1996; Viadero, 
1995). 
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OPTION B: We need to reform this way: 




The issue with this approach is that our reform mechanisms are not geared to take the 
longer view required for systemic reform, nor do most reformers possess the skill-sets, 
resources, nor frequently the patience for the long road. Systemic reform in a single 
school or department can be a ten year long, high-risk process. It is hard to scale up 
rapidly and is heavily dependent on a change agent for much of the time. Despite these 
huge issues there is a growing awareness that Option A is a dead end and that we need to 
at least start asking questions about Option B. 

Despite this somewhat pessimistic perspective about reform efforts, there exists a 
remarkable opportunity to design better programs that incorporate sound evaluation 
practice. We possess a powerful longstanding technology of teaching, learning and 
evaluation and a strong body of research on organizations and organizational behavior. 
We are right in the middle of an information revolution that offers us amazing tools to 
take care of the logistics of better program design and evaluation; and we have the 
historical precedent associated with the professional ascent of other fields as a guide. 
Collectively these assets should provide us with the toolbox to address the vexing 
questions we face in education related to program design and evaluation. 

In order to capitalize on these assets we first need to recognize that our enthusiasm for 
evaluation in education transcends our technology for designing and influencing the 
objects of our evaluation. We have three challenging prerequisite questions to answer: 
What is the design methodology that will allow schools and colleges to identify the 
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salient and tangible practices and processes that they will use to define their programs? 
How can we make a powerful technology of evaluation part of the design? How do we 
get those new designs and programs in place by reconciling the human and technical side 
of change in a strategic methodology for innovation (e.g., Evans, 1996; Fullan, 1997)? 

The thesis of this paper focuses on the first and second questions — the need to bring 
greater clarity to the core competencies and expected outcomes of programs in order to 
make evaluation purposeful. I have placed the emphasis here because of the need to first 
establish a clearer picture of what we can and should do. 

How do we move our definition of educational programs from those that focus on the 
perceived correlates of student success to those that are causally connected to that 
success? How do we create the clarity of program design and implementation that brings 
legitimacy to evaluation and more important enhances student learning? There are some 
straight forward albeit courageous steps that can be taken in order to make programmatic 
reform meaningful and evaluable. I will identify five key steps using examples derived 
from a longitudinal school reform project 

1 . Establish the Core Competency 

Establish the pedagogical and curricular core competency of the school or teacher 
education program. What practices does the school believe will influence the learning of 
students? How will it prepare faculty to deliver those practices and support them in their 
professional growth? For example, if a college professor is convinced that peer mediation 
should be taught in a methods course on teaching practice, then that faculty member 
should be able to train students to use it and should employ the practice in the course 
itself. As obvious as it seems, taking such a step requires a significant movement away 
from the prevailing journalistic approach to college teaching (the lecture including the 
meta-analysis of effects and if your lucky the classroom simulation) toward a model of 
clinically focused training. 

A core competency approach requires that mission and policy be articulated in practice. 
To extend the previous example for the purpose of illustration, in a core competency 
approach contemporary research on peer mediation is examined and the salient 
characteristics are identified. Those salient characteristics are incorporated into the 
position descriptions and roles and responsibilities of faculty. Training is provided to 
introduce and teach peer mediation techniques in addition to the provision of ongoing 
support and feedback about the use of the practice. Even the design of physical space 
may be informed by a clear definition of practice. 

What becomes possible when these “simple” steps are taken and then aggregated over all 
areas of competency is quite remarkable. First, and most important the secret of “best 
practice” is liberated for all to see and share or maybe more cynically the “emperor gets 
dressed.” 

— Meaningful evaluation becomes a possibility — 

Following are screen shots of a suite of evaluation tools that were developed based on a 
school’s definition of core competency in teaching, curriculum design, collaboration and 
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teamwork , professional growth and the use of technology. The tools have been in use for 
two years to build collaborative performance reports, gather and analyze surveys and 
conduct classroom observations over a school network. Teachers can go on-line to 
observe peers, complete surveys and receive and give feedback to peers and 
administrators as well as receive feedback from students. Over 6000 teacher evaluations 
by students have been gathered and analyzed over a two-year period. Each tool is based 
upon well-researched practice in the field; multiple evaluation approaches are used to tap 
the domains of interest and the perspectives of all stakeholders are included. Each 
stakeholder group is evaluated while also serving as evaluators. 



Classroom Observations 



Teacher Feedback 




School Tools Evaluation System 



Because the focus of the tools is the core competency, the professional lexicon associated 
with best practice becomes the domain of interest of all stakeholders. The questions on 
the student survey ask for specific feedback on student perceptions about their teachers’ 
demonstration of the core competency in each of the domains of interest. The items on 
self, peer and supervisory surveys are aligned with the student survey and the results 
triangulated with classroom observation and permanent product in the form of 
curriculum. The focus and clarity in the definition of the evaluation object raises the 
collective intelligence of the organization (Engelbart, 1998) by creating a common 
understanding of those things the community believes exert an influence on learning. 
Students, teachers and parents all learn about good practice because it is no longer a 
mystery. The organization builds institutional mastery and leverages what it values as 
best practice (Senge,1994). 
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Consider the rich dialogue that can occur in an evaluation process when teachers and 
students are talking about the most effective way to organize groups in a cooperative 
learning lesson, its place in a curriculum unit, or how to best recognize individuals and 
groups. Compare this with the “ has good rapport with students,” or “uses technology or 
teaching approaches effectively” type of feedback that so often characterizes the “drive 
by” evaluation that goes on in many educational settings where the essence of what 
constitutes effective practice remains undefined. 

Place yourself in the position of supervisor, teacher or student and consider also how 
much more productive a dialogue about collaboration and teamwork or the use of 
technology could be when those well researched characteristics of the approach are 
brought into the conversation through training, supportive feedback and valid evaluation 
mechanisms. From a teacher’s perspective the expectations are clear and the conversation 
authentic; from an administrators perspective the expectations for supervision are 
unambiguous and the interaction between teacher and supervisor based upon common 
understanding and lexicon of professional practice. For students the benefit is better 
learning and the opportunity to learn about their learning as they work with teachers who 
demonstrate a consistent and demonstrable understanding of their professional skill set 
(Bain, 1999). 

In the core competency approach good practice in teaching is not the only thing that is 
leveraged. Because the evaluation object is defined we can meaningfully leverage those 
best practices in evaluation, including multi-method approaches (e.g., ratings, 
observations, surveys, permanent products, collaborative conversation) that triangulate 
legitimate information from multiple sources (e.g., students, peers, supervisors, parents). 
In addition we leverage the use of information technology. The clarity of purpose allows 
us to build powerful tools that can address the well-documented logistical challenges 
associated with valid and timely performance appraisal. 

We also preclude the need for the magic “score.” The technology provides for easy 
triangulation and reporting allowing qualitative and quantitative data to remain in its 
original form. The tools build a powerful picture of performance based upon multiple 
perspectives and sources. The evaluation product is rich in perspective providing a 
validity that does not need to be reduced to a single statistical indicator. The information 
gathered serves not only as a basis for individual feedback but when aggregated across 
the school or program serves as a way to take the pulse of the institution in the areas 
associated with the core competency. 

2. Build a Defined Curriculum Model 

Build a curriculum model around the core competencies. Clearly quality programs are 
more than the aggregation of sets of best practices. The developmental course of 
curriculum and learner characteristics among other factors must be considered. However, 
the same research driven competency-based approach can be employed to build a 
curriculum model that integrates pedagogical approaches like cooperative learning into a 
developmental curriculum model. With such a model, not only is the sum of the parts 
evaluable, but the model itself. 

Following is an example of what is possible when the core competency associated with 
teaching and curriculum model is defined in a school setting. The screen shots are from a 
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suite of curriculum authoring tools that are used in a secondary school by faculty across 
disciplines to build and deliver curriculum. The tools are designed to integrate 
contemporary research on curriculum design including frameworks; authentic and 
portfolio assessment; (e.g., Wiggins, 1998, 1993); effective teaching, (e.g., Slavin, 1990; 
Rosenshine, 1986; Greenwood & Delquari, 1995); heterogeneous grouping (Wheelock, 
1992) and adapting instruction to deal with individual difference( Huck, Myers & 

Wilson, 1989) as well as multi-level instruction and classroom management. The tools 
translate the school’s core competency and curriculum design approach into a 
manageable design and delivery system for classroom use. They flatten the learning 
curve for faculty in their acquisition of knowledge in all areas associated with curriculum 
design and implementation by establishing a common lexicon of best practice and a 
common design methodology. And of course they reflect the items, methods and values 
described previously in the evaluation tools. 



Authentic Assessment r .„.„ i>i n „ 

Resource Lesson Design Tool 




Curriculum Authoring Tools’" 



Consider the implications of this example from a technological perspective. The powerful 
use of technology is only possible because the core competency is defined. The school 
has articulated what it believes to be best practice in assessment, teaching and curriculum 
design and in doing so has enabled the creation of a curriculum design tool that in turn 
leverages the core competency by translating it into manageable classroom practice. 
Instead of a generic lesson planner we have an authoring system infused with research on 
teaching and learning. 
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Build those structures that support the growth of faculty with the core competency. 

If we reflect on the necessary prerequisite conditions to implement the two pieces of 
software describe in the preceding examples we can see the absolute need for a systems 
approach to our thinking with regard to the design and evaluation of educational 
organizations. Neither would be possible to implement unless first they were connected 
together but even more important those mechanisms to train support and recognize 
excellence in their use need to be part of the organizational structure. Consider trying to 
implement the curriculum software using the linear model described in the opening to 
this paper. It would be “management suicide” to bring such a curriculum reform on board 
without also addressing those factors in the design of a school that establish, support and 
encourage the use of the model. 

Senge (1994) describes personal mastery and collaboration as essential characteristics of 
effective learning organizations. In our educational organizations we need to ask 
ourselves what kind of organizational characteristics will support individual mastery of 
the core competency and enable more than just the “heroes” to follow a path to better 
practice. This involves reconciling the personnel and professional growth structures in the 
organization with demonstrated excellence in the core competency, career, merit pay, 
faculty awards, portfolio assessment, integrated evaluation model, and collaborative 
organizational structures that devolve responsibility. Devolution becomes an exciting 
prospect when the individuals to whom authority is devolved have the skills in the core 
competency necessary to assume that responsibility. 

Following are two screen shots from an electronic portfolio completed by a faculty 
member as part of a career progression process. The portfolio is set up as an authentic 
record of excellence in all areas associated with the school’s model of curriculum and its 
core competency. Salary and career advancement is based on a demonstrated facility with 
the core competency, which develops over time. The portfolio also shows evidence of the 
connectedness necessary in the overall school design process. 

The curriculum model and evaluation tools described previously make the form of the 
portfolio possible. The definition of program makes the portfolio valid. The portfolio 
process focuses the organization’s attention on what it values and ensures that personal 
learning is central to the success of the school. 





Brewster Academy 
Professional Portfolio 
Bob Carter 
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4. Build a Technological Operating System 

Use technology to enable reform by translating the essential elements of the reforms into 
practical tools that teachers, students and parents can use. A secondary goal of this 
paper is to show how technology can be used to meaningfully empower change in 
schools when we use educational solutions for educational problems. In the cases of both 
the evaluation and curriculum tools, their value is based upon the extent to which their 
design is infused with best practice in fields related to educational systems. Both are part 
of an operating system for a school. The system takes the essential aspects of the school’s 
design and places them in a format that teachers and administrators use on a day to day 
basis to build curriculum, evaluate the program and interact with students. When the 
stakeholders use the tools they further the school’s vision and program. School 
improvement is practical. The tools translate the vision into a classroom reality and in 
doing so allow us to test the reform itself. The technology when used in this way forces 
us to answer the “what do you mean by that?” question, that is so often left unanswered 
in school reform. The technology forces us to consider the ergonomics of our innovation. 
The tools have to work in clear and simple ways if they are to serve faculty and you can 
guarantee the reformers will hear all about it if they do not (Cuban, 1993). 

5. Building the Connections 

Employ principles of effective practice to connect the teaching and learning processes. 
This paper has focussed on those process steps that can be taken to improve/design a 
learning organization. Making sophisticated connections between the learning process 
and the product is clearly the final piece in the puzzle. How do we build the kind of 
sophistication necessary to link the performance of the organization to the learning of 
students? The answer is, in exactly the same way and using the same methods described 
in building the curricular model and overall design of the school. “Quality learning to 
teach is the same as quality learning to learn.” This is a liberating idea. Those things that 
are important for student learning are equally compelling for our faculty. This includes 
authentic assessment, mastery learning, understanding our learning styles, collaboration 
and teamwork. In a true learning community we all benefit from a shared understanding 
of the learning process. We all use best practice. While faculty members employ personal 
portfolios to document the excellence in their teaching practice and career growth, 
students produce portfolios to graduate from the school using the same authentic 
assessment approach. Just as the evaluation tools described previously reflect those 
processes that are connected to quality teaching, so too do our measures of school related 
behavior that are predictive of student success in school, college and beyond. 

Earlier I talked about a departure from the “single score” as a measure of teacher 
performance. This concern with reductionist measures of teacher appraisal is analogous 
to the backlash regarding the preoccupation with standardized testing as the metric of 
school success for students. In the model described in this paper standardized test scores 
are part of the student evaluation picture, however they are triangulated with portfolio 
evaluation, measures of social growth and community participation that include student 
learning about how to learn. This evaluation is undertaken in ways that are similar to the 
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evaluation of teachers learning about teaching. In doing so the organization finds a 
consistent point of reconciliation around its learning priorities that are reflected in the 
processes, programs and practices that it adopts for the growth of both students and 
teachers. The ultimate goal may be to blur this distinction. 

Having spent the last eight years designing and implementing these processes it seems 
easy to roll them out in paragraph form. Suffice to say that each could constitute a 
position paper. However most important for the purposes of the present discussion is to 
emphasize the essential need for simultaneous attention to all aspects of a learning 
organization and the need for program clarity as a prerequisite to legitimate evaluation. 
This requires a systemic approach to the creation and reform of learning organizations. 

To make such an approach viable we need to take our models and ideas and try them out. 
This requires the engagement of research practitioners who are prepared to translate their 
models into practice. In doing so they must identify the associated core competencies, 
build tools for implementation and expend the energy necessary to build the human 
capital in schools and colleges that makes the rubber hit the road. None of this is easy. 
However one wonders what other choices we have after fifteen years of reform initiatives 
that have failed to make a sustainable difference in the core activity of schools beyond 
the idiosyncratic exemplar. 

In concluding I am reminded of the remarks of the late Grace Murray Hopper, Admiral, 
U.S.N (the first woman Admiral in the U.S. Navy, and the inventor of COBOL) who long 
before NIKE™ would -“just do it” -in the face of the many challenges she experienced as 
a computer pioneer. Maybe this is at least in part the answer to the third question raised 
earlier in this paper. When we can “just do it” with respect to the systemic reform of our 
learning organizations, evaluation will be easy. We just need to define “it” first. 
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Figure: 1 The Problem (With Pictures') 
Option A: In education we reform this way: 
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Figure:! The Problem (With Pictures') 
Option A: In education we reform this way: 
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Figure 2: Option B: We need to reform this way: 
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Figure :2 Option B: We need to reform this wav: 



OPTION B: We need 10 reform ihis way: 
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Figure 3: School Tools Evaluation System ™ 



18 

o 

ERIC 



Figure 3: School Tools Evaluation System ™ 
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Figure 4: Curriculum Authoring Tools ™ 
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Figure 4: Curriculum Authoring Tools ™ 
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Figure 5: Portfolio 
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Figure 5: Portfolio 
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